Don Lindberg's Outreach Legacy at the National Library of Medicine.

Friends and colleagues of Donald A.B. Lindberg M.D. came together to give tribute to his extraordinary contributions during his tenure (1984-2015) as Director of the U.S. National Library of Medicine (NLM). Dr. Lindberg died in 2019. The book, Transforming biomedical informatics and health information access: Don Lindberg and the U.S. National Library of Medicine. includes four sections. The ten edited chapters in section three (the Outreach section) are briefly summarized in this overview. As Associate Director for Health Information Programs Development, Elliot R. Siegel Ph.D. coordinated NLM's outreach programming under Dr. Lindberg's leadership from its inception in 1989 to his own retirement in 2010. Dr. Lindberg's legacy at NLM is one of new possibilities imagined, significant changes made in the mission and ethos of a venerable institution, and numerous successes achieved in a variety of settings and contexts. Like so much else Dr. Lindberg accomplished, these Outreach programs that profoundly changed the character of NLM would likely not have occurred without him. He made a difference.

Are We Making a Difference? Outreach Evaluation in Practice.

Donald A.B. Lindberg, M.D., brought with him when he joined NLM an inquisitive mind, tech savvy, and new ideas. He was an early advocate of both outreach and evaluation innovation at NLM. Dr. Lindberg initiated and supported multiple pilot test and implementation projects to strengthen NLM's health information outreach to healthcare providers, research scientists, health science and hospital librarians, and the general public, including minority and underserved populations. He helped steer NLM's transition to the Internet, and NLM's development of a robust framework for evaluating Internet and Web-based health information dissemination and outreach to its many audiences. Dr. Lindberg's leadership led to numerous landmark accomplishments, including the capacity-building "Measuring the Difference" outreach evaluation Guide, and a multidimensional approach to Internet and website evaluation that placed NLM at the forefront of federal agencies using these new and emerging technologies to support their missions.

Native Voices Exhibition: Stories of Health, Wellness, and Illness from American Indians, Alaska Natives, and Native Hawaiians.

The U.S. National Library of Medicine (NLM) exhibition known as Native Voices reflected Donald A.B. Lindberg M.D.'s keen and long-held desire to help improve public understanding of Native American health challenges and honor the culture, tradition, and healing ways of Native Peoples. A centerpiece of the exhibition was a large set of video interviews that Dr. Lindberg conducted with Native health and community leaders. Dr. Lindberg and his team engaged Native advisors in the exhibition development; sought Native input through Listening Circles, Tribal Consultations; and site visits, and made the video interviews accessible via interactive kiosks and iPads. For its time, this was state-of-the-art exhibition technology. The exhibition also included Native artifacts and art works to complement the videos, including a scale model of the iconic Hokule'a Native Hawaiian voyaging canoe, and a full-size Lummi Indian Healing Totem Pole. The totem journeyed across the U.S. prior to its installation next to the NLM herbal garden in Bethesda, MD. A traveling version of the exhibition visited more than 130 venues in 40 States across the U.S. The interview clips and other content are accessible on the exhibition website, and the full-length interviews are retained in the NLM permanent video collection.

Don Lindberg's outreach legacy at the National Library of Medicine.

Friends and colleagues of Donald A.B. Lindberg M.D. came together to give tribute to his extraordinary contributions during his tenure (1984-2015) as Director of the U.S. National Library of Medicine (NLM). Dr. Lindberg died in 2019. The book, Transforming biomedical informatics and health information access: Don Lindberg and the U.S. National Library of Medicine. includes four sections. The ten edited chapters in section three (the Outreach section) are briefly summarized in this overview. As Associate Director for Health Information Programs Development, Elliot R. Siegel Ph.D. coordinated NLM's outreach programming under Dr. Lindberg's leadership from its inception in 1989 to his own retirement in 2010. Dr. Lindberg's legacy at NLM is one of new possibilities imagined, significant changes made in the mission and ethos of a venerable institution, and numerous successes achieved in a variety of settings and contexts. Like so much else Dr. Lindberg accomplished, these Outreach programs that profoundly changed the character of NLM would likely not have occurred without him. He made a difference.

Native Voices Exhibition: Stories of health, wellness, and illness from American Indians, Alaska Natives, and Native Hawaiians.

The U.S. National Library of Medicine (NLM) exhibition known as Native Voices reflected Donald A.B. Lindberg M.D.'s keen and long-held desire to help improve public understanding of Native American health challenges and honor the culture, tradition, and healing ways of Native Peoples. A centerpiece of the exhibition was a large set of video interviews that Dr. Lindberg conducted with Native health and community leaders. Dr. Lindberg and his team engaged Native advisors in the exhibition development; sought Native input through Listening Circles, Tribal Consultations; and site visits, and made the video interviews accessible via interactive kiosks and iPads. For its time, this was state-of-the-art exhibition technology. The exhibition also included Native artifacts and art works to complement the videos, including a scale model of the iconic Hokule'a Native Hawaiian voyaging canoe, and a full-size Lummi Indian healing totem pole. The totem journeyed across the U.S. prior to its installation next to the NLM herbal garden in Bethesda, MD. A traveling version of the exhibition visited more than 130 venues in 40 States across the U.S. The interview clips and other content are accessible on the exhibition website, and the full-length interviews are retained in the NLM permanent video collection.

Are we making a difference? Outreach evaluation in practice.

Donald A.B. Lindberg, M.D., brought with him when he joined NLM an inquisitive mind, tech savvy, and new ideas. He was an early advocate of both outreach and evaluation innovation at NLM. Dr. Lindberg initiated and supported multiple pilot test and implementation projects to strengthen NLM's health information outreach to healthcare providers, research scientists, health science and hospital librarians, and the general public, including minority and underserved populations. He helped steer NLM's transition to the Internet, and NLM's development of a robust framework for evaluating Internet and Web-based health information dissemination and outreach to its many audiences. Dr. Lindberg's leadership led to numerous landmark accomplishments, including the capacity-building "Measuring the Difference" outreach evaluation Guide, and a multi-dimensional approach to Internet and website evaluation that placed NLM at the forefront of federal agencies using these new and emerging technologies to support their missions.

Enhancing the Capacity of Community Organizations to Evaluate HIV/AIDS Information Outreach: A Pilot Experiment in Expert Consultation.

The National Library of Medicine's AIDS Community Information Outreach Program (ACIOP) supports and enables access to health information on the Internet by community-based organizations. A technical assistance (TA) model was developed to enhance the capacity of ACIOP awardees to plan, evaluate, and report the results of their funded projects. This consisted of individual Consultation offered by an experienced evaluator to advise on the suitability of proposed project plans and objectives, improve measurement analytics, assist in problem resolution and outcomes reporting, and identify other improvement possibilities. Group webinars and a moderated blog for the exchange of project-specific information were also offered. Structured data collections in the form of reports, online surveys, and key informant telephone interviews provided qualitative feedback on project progress, satisfaction with the TA, and the perceived impact of the interventions on evaluation capacity building. The Model was implemented in the 2013 funding cycle with seven organizations, and the level of reported satisfaction was uniformly high. One-on-one TA was requested by four awardee organizations, and was determined to have made a meaningful difference with three. Participation in the webinars was mandatory and high overall; and was deemed to be a useful means for delivering evaluation information. In subsequent funding cycles, submission of a Logic Model will be required of awardees as a new model intervention in the expectation that it will produce stronger proposals, and enable the evaluation consultant to identify earlier intervention opportunities leading to project improvements and evaluation capacity enhancements.

Empowering Patients and Community Online: Evaluation of the AIDS Community Information Outreach Program.

The AIDS Community Information Outreach Program (ACIOP) was created in 1994 to assist the affected community in utilizing electronic HIV/AIDS information resources. Nearly 300 competitive awards have been made to mostly community-based organizations. A formal evaluation was undertaken to determine the performance and impact of the ACIOP. A mixed methods design combined quantitative abstractions and summarization of 47 awardee final reports from 44 organizations, and qualitative telephone interviews with 17 individuals representing 20 projects. Findings revealed that project objectives were mostly met; high-risk populations were reached; low resource organizations were funded; community partnerships were significant; projects built on existing efforts; information resources and training were tailored to local needs; and most projects overcame barriers experienced. Needed modifications centered on: 1) enhancing evaluation capacity at the individual project level and 2) revising project reporting requirements to increase the amount of information available to assess the ACIOP; both have been implemented.

NLM Workshop Marks 20 Years of Community Outreach and Capacity Building in HIV/AIDS.

The National Library of Medicine's (NLM) AIDS Community Information Outreach Program (ACIOP) was launched in 1994 to provide the HIV/AIDS affected community with access to vital health information resources increasingly becoming available on the Internet. Three hundred awards have been made mostly to community-based organizations. An evaluation in 2012 found that most program objectives are being met; a principal recommendation going forward was that NLM seek to enhance the capacity of community-based awardees to conduct evaluations of their own projects. This article reports on a workshop whose invitees were drawn from AIDS serving organizations, along with scientists, clinicians, and information technologists, to review the evaluation findings and recommendations. They considered alternatives for improving awardees' evaluation capabilities with the help of expert consultation, identified additional steps that could be taken to make individual project results more transparent and sharable, and looked at external influences ranging from mobile health devices to the latest HIV/AIDS scientific research findings that could be used to align future awards with unmet needs in the community. The paper identifies efforts subsequently made by ACIOP managers to prioritize and operationalize guidance from the evaluation and the workshop, and discusses the benefits of community engagement.

Defining the next generation journal: The NLM-Elsevier interactive publications experiment.

OBJECTIVE: A unique collaborative project to identify interactive enhancements to conventional-print journal articles, and to evaluate their contribution to readers' learning and satisfaction. HYPOTHESIS: It was hypothesized that (a) the enhanced article would yield more knowledge acquisition than the original article; (b) the interactivity aspects of the enhanced article would measurably contribute to the acquisition of knowledge; and (c) the enhancements to the original article would increase reader acceptance. METHODS: Fifteen SNMA medical students, assumed to have a greater generational familiarity and comfort level with interactive electronic media, reviewed 12 articles published in three Elsevier clinical and basic science journals. They used the Student National Medical Association's asynchronous online discussion forum over a four month period to suggest desired enhancements to improve learning. "Prognostic Factors in Stage T1 Bladder Cancer", published in the journal Urology was selected by the investigators as presenting the best opportunity to incorporate many of the students' suggested interactive and presentational enhancements in the limited timeframe available prior to the established test date. Educational, statistical, and medical consultants assisted in designing a test protocol in which 51 second to fourth year medical students were randomly assigned to experimental and control conditions, and were administered either the original or enhanced interactive version of the article on individual computer workstations. Test subjects consisted of 23 participants in the control group (8 males, 15 females) and 28 participants in the experimental group (9 males, 19 females). All subjects completed pre- and post-test instruments which measured their knowledge gain on 30 true-false and multiple-choice questions, along with 7 Likert-type questions measuring acceptance of the articles' format. Time to completion was recorded with the experimental group taking 22 min on average compared to 18 min for the controls; pre- and post-test times were 6 and 7 min, respectively. Statistical comparisons were based on change scores using either the Student t-test or the Two Way Analysis of Variance or Covariance. Significance was set at α = 0.05 or better. RESULTS: on the dependent measure of knowledge acquisition showed no difference overall on the 30 questions, but learning gain was statistically significant for the subset of 10 questions that measured gain on content that was accessible by the user-invoked interactive features of the enhanced article. Further analyses revealed significant interactions by student year and gender. Second year students (11 in the control group, 8 in the experimental group) were the best performers in terms of knowledge acquisition from both articles. The female medical students received a larger learning gain from journal enhancements and interactivity components than their male counterparts. Acceptance overall was greater for the experimental group who rated the experience more favorably than the controls. CONCLUSIONS: Failure to consider human factors such as gender and learning style may obscure underlying differences and their impact on the interactive aspects of scientific publications. Preliminary findings suggest the need for further study to include a heavier focus on interactivity apart from presentational enhancements; a more rigorous treatment of time as a specific variable; and an expanded experimental design that evaluates acquisition, understanding, integration and acceptance as dependent measures.

Web evaluation at the US National Institutes of Health: use of the American Customer Satisfaction Index online customer survey.

BACKGROUND: The National Institutes of Health (NIH), US Department of Health and Human Services (HHS), realized the need to better understand its Web users in order to help assure that websites are user friendly and well designed for effective information dissemination. A trans-NIH group proposed a trans-NIH project to implement an online customer survey, known as the American Customer Satisfaction Index (ACSI) survey, on a large number of NIH websites-the first "enterprise-wide" ACSI application, and probably the largest enterprise Web evaluation of any kind, in the US government. The proposal was funded by the NIH Evaluation Set-Aside Program for two years at a cost of US $1.5 million (US $1.275 million for survey licenses for 60 websites at US $18000 per website; US $225,000 for a project evaluation contractor). OBJECTIVE: The overall project objectives were to assess the value added to the participating NIH websites of using the ACSI online survey, identify any NIH-wide benefits (and limitations) of the ACSI, ascertain any new understanding about the NIH Web presence based on ACSI survey results, and evaluate the effectiveness of a trans-NIH approach to Web evaluation. This was not an experimental study and was not intended to evaluate the ACSI survey methodology, per se, or the impacts of its use on customer satisfaction with NIH websites. METHODS: The evaluation methodology included baseline pre-project websites profiles; before and after email surveys of participating website teams; interviews with a representative cross-section of website staff; observations of debriefing meetings with website teams; observations at quarterly trans-NIH Web staff meetings and biweekly trans-NIH leadership team meetings; and review and analysis of secondary data. RESULTS: Of the original 60 NIH websites signed up, 55 implemented the ACSI survey, 42 generated sufficient data for formal reporting of survey results for their sites, and 51 completed the final project survey. A broad cross-section of websites participated, and a majority reported significant benefits and new knowledge gained from the ACSI survey results. NIH websites as a group scored consistently higher on overall customer satisfaction relative to US government-wide and private sector benchmarks. CONCLUSIONS: Overall, the enterprise-wide experiment was successful. On the level of individual websites, the project confirmed the value of online customer surveys as a Web evaluation method. The evaluation results indicated that successful use of the ACSI, whether site-by-site or enterprise-wide, depends in large part on strong staff and management support and adequate funding and time for the use of such evaluative methods. In the age of Web-based e-government, a broad commitment to Web evaluation may well be needed. This commitment would help assure that the potential of the Web and other information technologies to improve customer and citizen satisfaction is fully realized.

MedlinePlus and the challenge of low health literacy: findings from the Colonias project.

OBJECTIVE: To explore the potential of a community-based health information outreach project to overcome problems associated with health literacy in low-income Hispanic communities along the Texas-Mexico border. METHODS: Using a train-the-trainer approach, community outreach workers known as promotoras were trained by a health information outreach team to search English and Spanish versions of MedlinePlus. These 15 promotoras submitted written examples on a weekly basis of the topics they helped residents explore on MedlinePlus and the ways in which the residents used the information. These weekly reports, along with verbal interviews with promotoras and others in the communities, allowed development of a database of 161 incidents ("stories") demonstrating how community residents used MedlinePlus. These stories were thematically analyzed to explore how the program benefited participants. RESULTS: The database of stories included examples of community residents becoming better informed about their illnesses, resolving to visit doctors, making decisions about recommended treatments, reducing their anxiety about health conditions, committing to healthy or preventive behavior, and assisting family members. CONCLUSION: With the help of paraprofessionals like promotoras, community-based health information outreach projects may improve the ability of community residents to understand their health conditions and to participate actively in their health care.

Information Rx: Evaluation of a new informatics tool for physicians, patients, and libraries.

This paper reports selected results from two comprehensive evaluation studies of the Information Prescription (or "Information Rx") Program implementation conducted from 2002-05 by the American College of Physicians Foundation (ACPF) and the U.S. National Library of Medicine (NLM). In this Program physicians are provided with Information Prescription pads, analogous to pads used to prescribe medications, that are used to direct patients to the MedlinePlus web site and its contents that are applicable to a patient's health condition. The results describe the Program's potential to enhance patient education and interpersonal communication from physician and patient perspectives. The findings suggest once physicians adopt the use of an information prescription, they perceive they are providing an additional clinical service that enhances patient education and interpersonal communication. For physicians, participation in information prescription may improve patient communication, encourage information seeking, and lessen the number of poor quality Internet searches that patients frequently self-perform and bring to a doctor's office. Similarly, once patients receive a recommendation from a physician to seek health information on the web, patients may be more comfortable with health seeking on the Internet and discussing their findings with their doctor. The conclusions of the two evaluation studies imply an Information Prescription fosters a dialogue between providers and patients, helps patients use the Internet more effectively and seems to favorably impact patient education. As the medical community and patient advocacy groups continue to emphasize the importance of evidence-based information as the gold standard for accepted care, it can be expected that informatics tools such as Information Rx will come to play an increasingly important role as a vehicle to help identify and access high quality health information on the Internet.

Health information Hispanic outreach in the Texas Lower Rio Grande Valley.

PURPOSE: This paper provides an overview of the two-year Texas Lower Rio Grande Valley Health Information Hispanic Outreach (HI HO) project. The project included a needs assessment, four pilot projects, and focus groups on the use of MedlinePlus and MedlinePlus en español. The needs assessment included a survey of physicians' information usage and a review of the circuit librarian program that had been established in 1989. The pilot projects were located at a high school, a rural health clinic, an urban health clinic, and a community center. Diffusion of innovation theory provided a framework for interpreting the results of the pilot projects. METHODS: The survey of physicians' information usage partially replicated a similar 1990 survey. The review of the circuit librarian program included usage statistics, interviews of administrators, and a survey of participants. Pilot project methodology varied by site. At the high school, four students were trained to instruct their peers in the use of MedlinePlus. At the two clinics, a computer workstation was installed for patients to access MedlinePlus. At the community center, staff were trained to use MedlinePlus en español to train community residents. Project evaluation included surveys, focus groups, and interviews. Indicators of success included increased level of consumer use of MedlinePlus, reports by key informants and consumers of how MedlinePlus was used, reports about training, and development of self-sustaining activity. RESULTS: The physician survey documented usage of health information resources in 2002 compared to 1990. The review of the circuit librarian program documented the change in program usage between 1989 and 2003. The pilot project at the high school was the most successful of the four pilot projects in introducing MedlinePlus to a large number of people, followed by the community center project. In the high school and community center projects, the participating institutions had reinforcing educational missions and paid staff who were highly motivated to achieve the project goals. The computer workstations projects at the two clinics were less successful, due in part to limited staff commitment and conflicting priorities. CONCLUSIONS: The HI HO project tested methods of reaching the Hispanic community in the Lower Rio Grande Valley region of Texas. The four HI HO pilot projects varied in achieving their stated objectives. But taken as a whole, the HI HO project significantly contributed to a better understanding of health information outreach to the Hispanic community, knowledge that should be useful to others with similar outreach activities.

The National Library of Medicine's Native American outreach portfolio: a descriptive overview.

OBJECTIVES: This paper provides the most complete accounting of the National Library of Medicine's (NLM's) Native outreach since 1995, when there were only a few scattered projects. METHOD: The descriptive overview is based on a review of project reports, inventories, and databases and input from the NLM Specialized Information Services Division, National Network Office of the Library Operations Division, National Network of Libraries of Medicine, and Office of Health Information Programs Development of the Office of the NLM Director. The overview focuses on NLM-supported or sponsored outreach initiatives involving Native peoples: American Indians, Alaska Natives, and Native Hawaiians. RESULTS: The review of NLM's relevant activities resulted in a portfolio of projects that clustered naturally into the following areas: major multisite projects: Tribal Connections and related, Native American Information Internship Project: Sacred Root, tribal college outreach and tribal librarianship projects, collaboration with inter-tribal and national organizations, participation in Native American Powwows, Native American Listening Circle Project, Native American Health Information, and other Native American outreach projects. IMPLICATIONS: NLM's Native American Outreach reached programmatic status as of late 2004. The companion paper identifies several areas of possible new or enhanced Native outreach activities. Both papers highlight the importance of solid reporting and evaluation to optimize project results and programmatic balance and priorities.

Assessment of the National Library of Medicine's health disparities plan: a focus on Native American outreach.

OBJECTIVES: Overcoming health disparities between majority and minority populations is a significant national challenge. This paper assesses outreach to Native Americans (American Indians, Alaska Natives, and Native Hawaiians) by the National Library of Medicine (NLM). A companion paper details NLM's portfolio of Native American outreach projects. METHOD: NLM's Native American outreach is assessed in light of the presentations at a community-based health information outreach symposium and the goals set by NLM's plan to reduce health disparities. RESULTS: NLM's current portfolio of Native American outreach projects appears most advanced in meeting the goal set in area 1 of the health disparities plan, "Promote use of health information by health professionals and the public." NLM's portfolio also shows significant strength and good progress regarding area 2 of the plan, "Expand partnerships among various types of libraries and community-based organizations." The portfolio is weaker in area 3, "Conduct and support informatics research." More knowledge-building efforts would benefit NLM, the National Network of Libraries of Medicine, and Native American and community-based organizations. IMPLICATIONS: The current Native American outreach portfolio should be continued, but new approaches are needed for evaluating Native American outreach and for forging collaborations with Native American groups, approaches grounded in consultation and mutual understanding of needs and perspectives.

Use of Internet audience measurement data to gauge market share for online health information services.

BACKGROUND: The transition to a largely Internet and Web-based environment for dissemination of health information has changed the health information landscape and the framework for evaluation of such activities. A multidimensional evaluative approach is needed. OBJECTIVE: This paper discusses one important dimension of Web evaluation-usage data. In particular, we discuss the collection and analysis of external data on website usage in order to develop a better understanding of the health information (and related US government information) market space, and to estimate the market share or relative levels of usage for National Library of Medicine (NLM) and National Institutes of Health (NIH) websites compared to other health information providers. METHODS: The primary method presented is Internet audience measurement based on Web usage by external panels of users and assembled by private vendors-in this case, comScore. A secondary method discussed is Web usage based on Web log software data. The principle metrics for both methods are unique visitors and total pages downloaded per month. RESULTS: NLM websites (primarily MedlinePlus and PubMed) account for 55% to 80% of total NIH website usage depending on the metric used. In turn, NIH.gov top-level domain usage (inclusive of NLM) ranks second only behind WebMD in the US domestic home health information market and ranks first on a global basis. NIH.gov consistently ranks among the top three or four US government top-level domains based on global Web usage. On a site-specific basis, the top health information websites in terms of global usage appear to be WebMD, MSN Health, PubMed, Yahoo! Health, AOL Health, and MedlinePlus. Based on MedlinePlus Web log data and external Internet audience measurement data, the three most heavily used cancer-centric websites appear to be www.cancer.gov (National Cancer Institute), www.cancer.org (American Cancer Society), and www.breastcancer.org (non-profit organization). CONCLUSIONS: Internet audience measurement has proven useful to NLM, with significant advantages compared to sole reliance on usage data from Web log software. Internet audience data has helped NLM better understand the relative usage of NLM and NIH websites in the intersection of the health information and US government information market sectors, which is the primary market intersector for NLM and NIH. However important, Web usage is only one dimension of a complete Web evaluation framework, and other primary research methods, such as online user surveys, usability tests, and focus groups, are also important for comprehensive evaluation that includes qualitative elements, such as user satisfaction and user friendliness, as well as quantitative indicators of website usage.

Tribal connections health information outreach: results, evaluation, and challenges.

In 1997, the National Library of Medicine (NLM), a component of the National Institutes of Health (NIH), initiated a program of intensified outreach to Native Americans, initially focusing on the Pacific Northwest in collaboration with the Pacific Northwest Regional Medical Library (PNRML). This initiative, known as the Tribal Connections Project, emphasized the establishment or strengthening of Internet connections at select Indian reservations and Alaska Native villages and related needs assessment and training. The hope was that these efforts would improve tribal access to health information available via the Internet and the Web. Phase I included sixteen tribal sites--eight in Washington, four in Alaska, two in Montana, and one each in Oregon and Idaho. Phase I results indicate that the project was successful in assessing local needs and building awareness of the Internet, forging new partnerships with and between the participating Indian reservations and Alaska Native villages and other organizations, making real improvements in the information technology (IT) infrastructure and Internet connectivity at fifteen of sixteen sites, and conducting training sessions with several hundred tribal participants across thirteen sites. Most importantly, the project demonstrated the key role of tribal community involvement and empowerment and contributed to development of an outreach evaluation field manual and the evolving concept of community-based outreach. The knowledge gained from Tribal Connections Project Phase I is helping refine and enhance subsequent NLM-sponsored tribal connections and similar community outreach efforts.